feat(document-search): add gcs source to the DocumentMeta #25

akotyla · 2024-09-19T09:13:26Z

No description provided.

github-actions · 2024-09-19T09:17:44Z

Trivy scanning results.

.venv/lib/python3.10/site-packages/PyJWT-2.9.0.dist-info/METADATA (secrets)

Total: 1 (MEDIUM: 1, HIGH: 0, CRITICAL: 0)

MEDIUM: JWT (jwt-token)
════════════════════════════════════════
JWT token
────────────────────────────────────────
.venv/lib/python3.10/site-packages/PyJWT-2.9.0.dist-info/METADATA:80
────────────────────────────────────────
78 >>> encoded = jwt.encode({"some": "payload"}, "secret", algorithm="HS256")
79 >>> print(encoded)
80 [ *********************************************************************************************************
81 >>> jwt.decode(encoded, "secret", algorithms=["HS256"])
────────────────────────────────────────

.venv/lib/python3.10/site-packages/litellm/proxy/_types.py (secrets)

Total: 1 (MEDIUM: 1, HIGH: 0, CRITICAL: 0)

MEDIUM: Slack (slack-web-hook)
════════════════════════════════════════
Slack Webhook
────────────────────────────────────────
.venv/lib/python3.10/site-packages/litellm/proxy/_types.py:1287
────────────────────────────────────────
1285 alert_to_webhook_url: Optional[Dict] = Field(
1286 None,
1287 [ description="Mapping of alert type to webhook url. e.g. alert_to_webhook_url: {'budget_alerts': '*****************************************************************************'}",
1288 )
────────────────────────────────────────

github-actions · 2024-09-19T09:19:23Z

🧪 Test coverage: 91.89%

Code Coverage Summary

Filename                                                                                                Stmts    Miss  Cover    Missing
----------------------------------------------------------------------------------------------------  -------  ------  -------  --------------------------------
packages/ragbits-core/src/ragbits/core/__init__.py                                                          0       0  100.00%
packages/ragbits-core/src/ragbits/core/embeddings/__init__.py                                               0       0  100.00%
packages/ragbits-core/src/ragbits/core/embeddings/base.py                                                   4       0  100.00%
packages/ragbits-core/src/ragbits/core/llms/__init__.py                                                     4       0  100.00%
packages/ragbits-core/src/ragbits/core/llms/base.py                                                        33       4  87.88%   33, 52, 88, 97
packages/ragbits-core/src/ragbits/core/llms/litellm.py                                                     25       4  84.00%   8-9, 54, 85
packages/ragbits-core/src/ragbits/core/llms/local.py                                                       24      10  58.33%   8-9, 42-47, 57, 70-71
packages/ragbits-core/src/ragbits/core/llms/types.py                                                        8       2  75.00%   24, 28
packages/ragbits-core/src/ragbits/core/llms/clients/__init__.py                                             4       0  100.00%
packages/ragbits-core/src/ragbits/core/llms/clients/base.py                                                23       0  100.00%
packages/ragbits-core/src/ragbits/core/llms/clients/exceptions.py                                          14       6  57.14%   7-8, 17, 26-27, 36
packages/ragbits-core/src/ragbits/core/llms/clients/litellm.py                                             50      10  80.00%   10-11, 70, 108, 122-127
packages/ragbits-core/src/ragbits/core/llms/clients/local.py                                               37      12  67.57%   11-12, 62-70, 92-103
packages/ragbits-core/src/ragbits/core/prompt/__init__.py                                                   2       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/base.py                                                      18       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/parsers.py                                                   34       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/prompt.py                                                    96       2  97.92%   103, 107
packages/ragbits-core/tests/unit/__init__.py                                                                0       0  100.00%
packages/ragbits-core/tests/unit/llms/__init__.py                                                           0       0  100.00%
packages/ragbits-core/tests/unit/llms/test_litellm.py                                                      63       0  100.00%
packages/ragbits-core/tests/unit/prompts/__init__.py                                                        0       0  100.00%
packages/ragbits-core/tests/unit/prompts/test_parsers.py                                                   65       0  100.00%
packages/ragbits-core/tests/unit/prompts/test_prompt.py                                                    81       0  100.00%
packages/ragbits-dev-kit/src/ragbits/dev_kit/__init__.py                                                    0       0  100.00%
packages/ragbits-dev-kit/src/ragbits/dev_kit/prompt_lab/__init__.py                                         0       0  100.00%
packages/ragbits-dev-kit/src/ragbits/dev_kit/prompt_lab/discovery/__init__.py                               2       0  100.00%
packages/ragbits-dev-kit/src/ragbits/dev_kit/prompt_lab/discovery/prompt_discovery.py                      38       1  97.37%   62
packages/ragbits-dev-kit/tests/unit/discovery/prompt_classes_for_tests.py                                  27       0  100.00%
packages/ragbits-dev-kit/tests/unit/discovery/test_prompt_discovery.py                                     25       0  100.00%
packages/ragbits-dev-kit/tests/unit/discovery/ragbits_tests_pkg_with_prompts/__init__.py                    2       0  100.00%
packages/ragbits-dev-kit/tests/unit/discovery/ragbits_tests_pkg_with_prompts/prompts/__init__.py            3       0  100.00%
packages/ragbits-dev-kit/tests/unit/discovery/ragbits_tests_pkg_with_prompts/prompts/temp_prompt1.py       14       0  100.00%
packages/ragbits-dev-kit/tests/unit/discovery/ragbits_tests_pkg_with_prompts/prompts/temp_prompt2.py       14       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/__init__.py                                    2       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/_main.py                                      34       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/documents/__init__.py                          0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/documents/document.py                         55       3  94.55%   52, 95, 124
packages/ragbits-document-search/src/ragbits/document_search/documents/element.py                          30       1  96.67%   34
packages/ragbits-document-search/src/ragbits/document_search/documents/sources.py                          45       8  82.22%   13-14, 58, 87, 106, 109, 120-121
packages/ragbits-document-search/src/ragbits/document_search/ingestion/__init__.py                          0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/document_processor.py               21       1  95.24%   80
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/__init__.py                0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/base.py                   14       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/dummy.py                  11       1  90.91%   29
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured.py           26       4  84.62%   79-89, 93
packages/ragbits-document-search/src/ragbits/document_search/retrieval/__init__.py                          0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/__init__.py               0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/base.py                   5       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/noop.py                   5       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/__init__.py                0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/base.py                    6       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/noop.py                    7       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/vector_store/__init__.py                       0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/vector_store/base.py                          12       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/vector_store/in_memory.py                     18       0  100.00%
packages/ragbits-document-search/tests/__init__.py                                                          0       0  100.00%
packages/ragbits-document-search/tests/helpers.py                                                           3       0  100.00%
packages/ragbits-document-search/tests/integration/__init__.py                                              0       0  100.00%
packages/ragbits-document-search/tests/integration/test_unstructured.py                                    38      24  36.84%   22-29, 37-43, 51-57, 65-72
packages/ragbits-document-search/tests/unit/__init__.py                                                     0       0  100.00%
packages/ragbits-document-search/tests/unit/test_document_processor.py                                     11       0  100.00%
packages/ragbits-document-search/tests/unit/test_document_search.py                                        14       0  100.00%
packages/ragbits-document-search/tests/unit/test_documents.py                                              13       0  100.00%
packages/ragbits-document-search/tests/unit/test_elements.py                                               15       0  100.00%
packages/ragbits-document-search/tests/unit/test_gcs_source.py                                             14       0  100.00%
packages/ragbits-document-search/tests/unit/test_providers.py                                              22       0  100.00%
packages/ragbits-document-search/tests/unit/test_simple_vector_store.py                                    16       0  100.00%
TOTAL                                                                                                    1147      93  91.89%

Diff against main

Filename                                                                             Stmts    Miss  Cover
---------------------------------------------------------------------------------  -------  ------  --------
packages/ragbits-document-search/src/ragbits/document_search/documents/sources.py      +29      +7  -11.53%
packages/ragbits-document-search/tests/unit/test_gcs_source.py                         +14       0  +100.00%
TOTAL                                                                                  +43      +7  -0.32%

Results for commit: 366aade

Minimum allowed coverage is 60%

♻️ This comment has been updated with latest results

packages/ragbits-document-search/src/ragbits/document_search/documents/sources.py

mhordynski · 2024-09-19T15:13:21Z

packages/ragbits-document-search/src/ragbits/document_search/documents/sources.py

+    An object representing a GCS file source.
+    """
+
+    source_type: Literal["google_cloud_storage_file"] = "google_cloud_storage_file"


gcs will be enough

ok, changed

Literally gcs :D

packages/ragbits-document-search/pyproject.toml

packages/ragbits-document-search/src/ragbits/document_search/documents/sources.py

mhordynski

lgtm

…ocuments/sources.py Co-authored-by: Mateusz Hordyński <[email protected]>

akotyla added 2 commits September 18, 2024 14:39

Initial version of GCS source

ddfda9a

Changes in the method

ac8b000

akotyla added feature New feature or request document search Changes to the document search package labels Sep 19, 2024

akotyla requested a review from mhordynski September 19, 2024 09:13

akotyla self-assigned this Sep 19, 2024

mhordynski reviewed Sep 19, 2024

View reviewed changes

Changes after review

4281382

mhordynski reviewed Sep 23, 2024

View reviewed changes

packages/ragbits-document-search/src/ragbits/document_search/documents/sources.py Show resolved Hide resolved

mhordynski reviewed Sep 23, 2024

View reviewed changes

packages/ragbits-document-search/src/ragbits/document_search/documents/sources.py Show resolved Hide resolved

mhordynski linked an issue Sep 23, 2024 that may be closed by this pull request

feat(document-search): add gcs source to the DocumentMeta #6

Closed

akotyla added 4 commits September 24, 2024 12:38

Fixes after review

e610fb9

Fix tests

5dffc9b

Resolve conflict

d097248

More fixes

3798a54

mhordynski reviewed Sep 24, 2024

View reviewed changes

packages/ragbits-document-search/src/ragbits/document_search/documents/sources.py Outdated Show resolved Hide resolved

mhordynski approved these changes Sep 24, 2024

View reviewed changes

Update packages/ragbits-document-search/src/ragbits/document_search/d…

366aade

…ocuments/sources.py Co-authored-by: Mateusz Hordyński <[email protected]>

akotyla merged commit 584fd93 into main Sep 25, 2024
3 checks passed

mhordynski deleted the ak/add-gcs-support branch November 21, 2024 21:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(document-search): add gcs source to the DocumentMeta #25

feat(document-search): add gcs source to the DocumentMeta #25

akotyla commented Sep 19, 2024

github-actions bot commented Sep 19, 2024 •

edited

Loading

github-actions bot commented Sep 19, 2024 •

edited

Loading

mhordynski Sep 19, 2024

akotyla Sep 23, 2024

mhordynski Sep 23, 2024

mhordynski left a comment

feat(document-search): add gcs source to the DocumentMeta #25

feat(document-search): add gcs source to the DocumentMeta #25

Conversation

akotyla commented Sep 19, 2024

github-actions bot commented Sep 19, 2024 • edited Loading

.venv/lib/python3.10/site-packages/PyJWT-2.9.0.dist-info/METADATA (secrets)

.venv/lib/python3.10/site-packages/litellm/proxy/_types.py (secrets)

github-actions bot commented Sep 19, 2024 • edited Loading

Code Coverage Summary

Diff against main

mhordynski Sep 19, 2024

Choose a reason for hiding this comment

akotyla Sep 23, 2024

Choose a reason for hiding this comment

mhordynski Sep 23, 2024

Choose a reason for hiding this comment

mhordynski left a comment

Choose a reason for hiding this comment

github-actions bot commented Sep 19, 2024 •

edited

Loading

github-actions bot commented Sep 19, 2024 •

edited

Loading